Note, all of the patents, patent applications, technical papers and other references referenced below are incorporated herein by reference in their entirety unless stated otherwise.
Automobiles equipped with airbags are well known in the prior art. In such airbag systems, the car crash is sensed and the airbags rapidly inflated thereby insuring the safety of an occupation in a car crash. Many lives have now been saved by such airbag systems. However, depending on the seated state of an occupant, there are cases where his or her life cannot be saved even by present airbag systems. For example, when a passenger is seated on the front passenger seat in a position other than a forward facing, normal state, e.g., when the passenger is out of position and near the deployment door of the airbag, there will be cases when the occupant will be seriously injured or even killed by the deployment of the airbag.
Also, sometimes a child seat is placed on the passenger seat in a rear facing position and there are cases where a child sitting in such a seat has been seriously injured or killed by the deployment of the airbag.
Furthermore, in the case of a vacant seat, there is no need to deploy an airbag, and in such a case, deploying the airbag is undesirable due to a high replacement cost and possible release of toxic gases into the passenger compartment. Nevertheless, most airbag systems will deploy the airbag in a vehicle crash even if the seat is unoccupied.
Thus, whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and over 100 people have now been killed. Thus, significant improvements need to be made to airbag systems. As discussed in detail in U.S. Pat. No. 5,653,462, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of the deployment thereof. Also, a child in a rear facing child seat that is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. “How Airbags Work” presented at the International Conference on Seatbelts and Airbags in 1993 in Canada, occupant position sensing and rear facing child seat detection systems are required in order to minimize the damages caused by deploying front and side airbags. It also may be required in order to minimize the damage caused by the deployment of other types of occupant protection and/or restraint devices that might be installed in the vehicle.
For these reasons, there has been proposed an occupant sensor system also known as a seated-state detecting unit such as disclosed in the following U.S. patents assigned to the current assignee of the present application: Breed et al. (U.S. Pat. No. 5,563,462); Breed et al. (U.S. Pat. No. 5,829,782); Breed et al. (U.S. Pat. No. 5,822,707): Breed et al. (U.S. Pat. No. 5,694,320); Breed et al. (U.S. Pat. No. 5,748,473); Varga et al. (U.S. Pat. No. 5,943,295); Breed et al. (U.S. Pat. No. 6,078,854); Breed et al. (U.S. Pat. No. 6,081,757); and Breed et al. (U.S. Pat. No. 6,242,701). Typically, in some of these designs three or four sensors or sets of sensors are installed at three or four points in a vehicle for transmitting ultrasonic or electromagnetic waves toward the passenger or drivers seat and receiving the reflected waves. Using appropriate hardware and software, the approximate configuration of the occupancy of either the passenger or driver seat can be determined thereby identifying and categorizing the occupancy of the relevant seat.
These systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded and unwanted airbag deployments when a front seat is unoccupied. Some of the airbag systems will also protect rear seat occupants in vehicle crashes and all occupants in side impacts.
However, there is a continual need to improve the systems which detect the presence of occupants, determine if they are out-of-position and to identify the presence of a rear facing child seat in the rear seat as well as the front seat. Future automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly. The improvements described below minimize this cost by not deploying an airbag for a seat, which is not occupied by a human being. An occupying item of a seat may be a living occupant such as a human being or dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
A child in a rear facing child seat, which is placed on the right front passenger seat, is in danger of being seriously injured if the passenger airbag deploys. This has now become an industry-wide concern and the U.S. automobile industry is continually searching for an economical solution that will prevent the deployment of the passenger side airbag if a rear facing child seat is present. The inventions disclosed herein include sophisticated apparatus to identify objects within the passenger compartment and address this concern.
The need for an occupant out-of-position sensor has also been observed by others and several methods have been described in certain U.S. patents for determining the position of an occupant of a motor vehicle. However, none of these prior art systems are capable of solving the many problems associated with occupant sensors and no prior art has been found that describe the methods of adapting such sensors to a particular vehicle model to obtain high system accuracy. Also, none of these systems employ pattern recognition technologies that are believed to be essential to accurate occupant sensing. Each of these prior are systems will be discussed below.
In 1984, the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation issued a requirement for frontal crash protection of automobile occupants known as FMVSS-208. This regulation mandated “passive occupant restraints” for all passenger cars by 1992. A further modification to FMVSS-208 required both driver and passenger side airbags on all passenger cars and light trucks by 1998. FMVSS-208 was later modified to require all vehicles to have occupant sensors. The demand for airbags is constantly accelerating in both Europe and Japan and all vehicles produced in these areas and eventually worldwide will likely be, if not already, equipped with airbags as standard equipment and eventually with occupant sensors.
A device to monitor the vehicle interior and identify its contents is needed to solve these and many other problems. For example, once a Vehicle Interior Identification and Monitoring System (VIMS) for identifying and monitoring the contents of a vehicle is in place, many other products become possible as discussed below.
Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident. The VIMS discussed in U.S. Pat. No. 5,829,782 will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. The inventions here are improvements on that VIMS system and some use an advanced optical system comprising one or more CCD or CMOS arrays plus a source of illumination preferably combined with a trained neural network pattern recognition system.
In the early 1990's, the current assignee (ATI) developed a scanning laser radar optical occupant sensor that had the capability of creating a three dimensional image of the contents of the passenger compartment. After proving feasibility, this effort was temporarily put aside due to the high cost of the system components and the current assignee then developed an ultrasonic based occupant sensor that was commercialized and is now in production on some Jaguar models. The current assignee has long believed that optical systems would eventually become the technology of choice when the cost of optical components came down. This has now occurred and for the past several years, ATI has been developing a variety of optical occupant sensors.
The current assignee's first camera optical occupant sensing system was an adult zone-classification system that detected the position of the adult passenger. Based on the distance from the airbag, the passenger compartment was divided into three zones, namely safe-seating zone, at-risk zone, and keep-out zone. This system was implemented in a vehicle under a cooperative development program with NHTSA. This proof-of-concept was developed to handle low-light conditions only. It used three analog CMOS cameras and three near-infrared LED clusters. It also required a desktop computer with three image acquisition boards. The locations of the camera/LED modules were: the A-pillar, the IP, and near the overhead console. The system was trained to handle camera blockage situations, so that the system still functioned well even when two cameras were blocked. The processing speed of the system was close to 50 fps giving it the capability of tracking an occupant during pre-crash braking situations—that is a dynamic system.
The second camera optical system was an occupant classification system that separated adult occupants from all other situations (i.e., child, child restraint and empty seat). This system was implemented using the same hardware as the first camera optical system. It was also developed to handle low-light conditions only. The results of this proof-of-concept were also very promising.
Since the above systems functioned well even when two cameras were blocked, it was decided to develop a stand alone system that is FMVSS208-compliant, and price competitive with weight-based systems but with superior performance. Thus, a third camera optical system (for occupant classification) was developed. Unlike the earlier systems, this system used one digital CMOS camera and two high-power near-infrared LEDs. The camera/LED module was installed near the overhead console and the image data was processed using a laptop computer. This system was developed to divide the occupancy state into four classes: 1) adult; 2) child, booster seat and forward facing child seat; 3) infant carrier and rearward facing child seat; and 4) empty seat. This system included two subsystems: a nighttime subsystem for handling low-light conditions, and a daytime subsystem for handling ambient-light conditions. Although the performance of this system proved to be superior to the earlier systems, it exhibited some weakness mainly due to a non-ideal aiming direction of the camera.
Finally, a fourth camera optical system was implemented using near production intent hardware using, for example, an ECU (Electronic Control Unit) to replace the laptop computer. In this system, the remaining problems of earlier systems were overcome. The hardware in this system is not unique so the focus below will be on algorithms and software which represent the innovative heart of the system.